<HTML>
<BODY BGCOLOR="#FFFFFF">

Classes supporting the monitoring and filtering of IP packets processed
by the IP protocol (SSF.OS.IP), for the collection of IP flow data
from the routers in the spirit of Cisco's FlowCollector. In 1.2.8, NetFlow/Filter/FlowFilter.java is removed, which caused problem in jdk1.4.0 beta version.

<H1>
SSF Implementation of NetFlow - Ver1.2.8</H1>

<HR WIDTH="100%">
<H3>
Contents</H3>
<A HREF="#updates">Updates</A>
<BR><A HREF="#background">Background of NetFlow</A>
<BR><A HREF="#Implementation">NetFlow implementation in SSF</A>
<BR><A HREF="#Data format">NetFlow data format</A>
<BR><A HREF="#Configuring monitors">Configuring monitors in DML</A>
<BR><A HREF="#Examples and tests">Examples and tests</A>
<BR><A HREF="#Download">Download</A>
<BR><A HREF="#References">References</A>
<BR>
<HR WIDTH="100%">

<!-----------  Updates ------------------------->
<H3>
<A NAME="updates"></A>Updates</H3>
For version 1.2.5:
<UL>
<LI>An extended version of IpFlowCollector: IpFlowCollectorWD is added. It enable user to have some notion of "domain". Basically, the hosts that have the same NHI prefix will be deemed as in a single domain. How long the prefix is for a domain is optional.
<LI>IpMonitor (in SSF/OS) is changed to ProtocolMonitor to enable more general purpose usage.
<LI>Package re-organized to make it look more clean, several small gadgets added in the Util direcotry. 
</UL>

<!--------------------- background --------------------->
<H3>
<A NAME="background"></A>Background of NetFlow</H3>
"NetFlow" here basically refers to the <I>continuous</I> traffic between
two <I>entities</I> on the Internet. To be more specific, the continuity
means the time interval between any two packets is smaller than some threashhold,
and "entities" here may refer to hosts or routers or even subnets. The
data that can characterize the traffic is called NetFlow data. Collecting
and analyzing the NetFlow data can provide important information for network
planning, monitoring, and a lot of more other work.

<!--------------------- Implementation ------------------>
<H3>
<A NAME="Implementation"></A>NetFlow implementation in SSF</H3>
In the blueprint of NetFlow work in SSF, there are two main parts, collecting
NetFlow data from the simulated network and analyzing NetFlow data. This
document is mainly about the data collecting.
<P>The main idea is to set up mulitple monitors on the simulated network
to collect NetFlow data. The monitors are specified with DML (more detail
in "Configuring monitors in DML" section.). There are some "pipes" for
these monitos to dump NetFlow data. In fact, the monitors in the same timeline
will share the same pipe. The other end of the pipe could be files or remote
machines, where the NetFlow data is stored.
<P>Take a closer look at how the NetFlow data is collected on a "host".&nbsp;
In the protocol graph of a host, there is a new protocol session called
"ProbeSession". It's used to provide the interface for the "monitor" to
dump data. The NetFlow data is collected through an event-driven method.
The protocol session that a monitor is monitoring has a handle of the monitor.
It calls the monitor when some events of interest happen. The monitor extracts
the information needed and stored it in a "conOpenTable" (Connection-Open-Table).
Periodically, the monitor moves the data of those finished flow to another
table, from where it will be dumped to the pipe provided by "ProbeSession".
<P>This method is actually a general method in network monitoring. As you
can see, there is absolutely no restriction of what the monitor will extract
and what it will do. By writing the user's customized monitor component,
the user can collect the data of his/her own interest.
<P>To write the user's customized monitor, the user must understand how
the monitor components are pluged into the protocol graph or a host/router.
The explanation below only uses IP and ProtocolMonitor as examples. All the monitors
for the same protocol have the same interface. For IP, the interface is
as below:
<CENTER><TABLE BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<TR>
<TD>public interface ProtocolMonitor
<BR>{
<BR>&nbsp;&nbsp;&nbsp; /** the api used to filter the packet */
<BR>&nbsp;&nbsp;&nbsp; void receive(ProtocolMessage packet, ProtocolSession fromSession, ProtocolSession toSession);
<P>&nbsp;&nbsp;&nbsp; /** config this protocolsession */
<BR>&nbsp;&nbsp;&nbsp; void config(ProtocolMonitor owner, Configuration cfg) throws
configException;
<P>&nbsp;&nbsp;&nbsp; /** other init work that will be done in the "init"
phase */
<BR>&nbsp;&nbsp;&nbsp; void init();
<BR>}</TD>
</TR>
</TABLE></CENTER>
The IP is slightly revised to support its monitor option, and the DML configuration
of the revised IP is shown below in the "Configurnig Monitor" section.
In the code of revised IP <I>config()</I> function, it calls a new member
function <I>createMonitor()</I> when it finds out that the Monitor option
is activated in the DML. A handle of the <I>ProtocolMonitor</I> created will
be saved. When the revised IP finished routine initialization, it also
calls the <I>init()</I> or the ProtocolMonitor to complete initializing the monitor.
Another change of IP is in the <I>push</I> function, and it wakes up the
ProtocolMonitor when appropriate events happen. In the implementation now, the
<I>receive</I>
function of Monitor is not invoked when the packet exhausts its TTL and got dropped. (This may be changed as time
goes on.)
<P>In order to minimize the impact of introducting monitors to the simulation,
it's critical to have fast operation when insert, delete and retrive the
flow data in the flow table. <I>ShrinkableHashMap</I> is implemented for
this purpose. It has the following characteristics:
<UL>
<LI>
It can dynamically increase/decrease its size according to the utilization.</LI>

<LI>
Fast insert, delete and retrieval of items inside because it's a "hashMap".</LI>

<LI>
The operations are all unsynchronized.</LI>
</UL>

<H3>
<A NAME="Data format"></A>NetFlow data format</H3>
In the real world, the Cisco routers also provide the service of collecting
and analyze NetFlow data. The user specify some machine to store the data,
and the routers collecting the NetFlow data will send it to that machine
with UDP protocol. There are several versions of the NetFlow data of Cisco
routers [1], and below is a comparation between version7 Cisco NetFlow
data format and the data format in SSF implementation.
<P><B>Common fields</B>
<CENTER><TABLE BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<TR>
<TD>src_addr</TD>

<TD>Source IP address of the NetFlow (If there is src_mask, then it's the
address after applying the mask.)</TD>
</TR>

<TR>
<TD>dst_addr</TD>

<TD>Destination IP address of the NetFlow (If there is dst_mask, then it's
the address after applying the mask.)</TD>
</TR>

<TR>
<TD>input</TD>

<TD>The NIC from which the traffic is coming in. In Cisco NetFlow ver7.0,
it's always set to be 0.</TD>
</TR>

<TR>
<TD>output</TD>

<TD>The NIC from which the traffic is going out.</TD>
</TR>

<TR>
<TD>nextHop</TD>

<TD>The next hop IP. In cisco NetFlow7.0, it's always set to be 0.</TD>
</TR>

<TR>
<TD>First</TD>

<TD>The system time of receiving the first packet of this flow.</TD>
</TR>

<TR>
<TD>Last</TD>

<TD>The system time of receiving the last packet of this flow.</TD>
</TR>

<TR>
<TD>dPkts</TD>

<TD>Number of packets in this flow by now.</TD>
</TR>

<TR>
<TD>dOctets</TD>

<TD>Number of bytes of data (above 3 layer) in this flow by now</TD>
</TR>

<TR>
<TD>tcp_flag</TD>

<TD>The cumulative OR of tcp flags, if any.</TD>
</TR>

<TR>
<TD>protocol</TD>

<TD>The protocol type of the traffic that this netflow is about.</TD>
</TR>
</TABLE></CENTER>

<P><B>Different fields</B>
<P><I>In Cisco ver7.0</I>
<CENTER><TABLE BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<TR>
<TD>srcport</TD>

<TD>TCP/UDP source port number, set to 0 if the flow mask is destination-only
or source-destination.</TD>
</TR>

<TR>
<TD>dstport</TD>

<TD>TCP/UDP destination port number, set to 0 if the flow mask is destination-only
or source-destination.</TD>
</TR>

<TR>
<TD>flags</TD>

<TD>Flags indicating, among other things, what fields are invalid</TD>
</TR>

<TR>
<TD>tos</TD>

<TD>IP type of service, it's set to be the ToS of the first packet of this
flow.</TD>
</TR>

<TR>
<TD>src_as</TD>

<TD>Source autonamous system number, either origin or peer, always set
to be 0.</TD>
</TR>

<TR>
<TD>dst_as</TD>

<TD>Destination aumonumous system number, either origin or peer, always
set to be 0.</TD>
</TR>

<TR>
<TD>src_mask</TD>

<TD>Source address <B>prefix</B> mask, always set to be zero.</TD>
</TR>

<TR>
<TD>dst_mask</TD>

<TD>Destination address <B>prefix</B> mask, always set to be zero.&nbsp;</TD>
</TR>

<TR>
<TD>router_sc</TD>

<TD>IP address of the router that is bypassed by the Catalyst 5000 series
switch. This is the same address the router uses when it sends NetFlow
export packets.&nbsp;</TD>
</TR>
</TABLE></CENTER>

<P><I>In SSF implementation</I>
<CENTER><TABLE BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<TR>
<TD>nhi</TD>

<TD>The NHI address of the router that collects the flow data. Functionally
it's the same as "router_sc" in Cisco ver7.0</TD>
</TR>

<TR>
<TD>src_mask</TD>

<TD>Source address <B>suffix</B> mask. It's used so that some pre-aggregation
can be done in SSF simulation.</TD>
</TR>

<TR>
<TD>dst_mask</TD>

<TD>Destination address <B>suffix</B> mask. Similar to src_maks, it's used
to help pre-aggregation.</TD>
</TR>

<TR>
<TD>inputType</TD>
<TD>Type of the input port: flow came in from an external host or an internal one. (for IpFlowCollectorWD)</TD>
</TR>

<TR>
<TD>outputType</TD>
<TD>Type of the output port: flow forwarded to an external host or an internal one. (for IpFlowCollectorWD)</TD>
</TR>
 
</TABLE></CENTER>

<H3>
<A NAME="Configuring monitors"></A>Configuring monitors in DML</H3>
This section is a tutorial of how to specify monitors with DML in the SSF
network configuration file. For each protocol (or NIC), its own monitor
could be specified with DML. To help explanation, <I>IP</I> and <I>ProtocolMonitor</I>
will be used as examples below.
<P>In IP protocolSession, an attribute "<I>Monitor</I>" will be added.
The following is an example of configuring IP with DML.
<CENTER><TABLE BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<TR>
<TD>&nbsp;ProtocolSession [
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name ip use SSF.OS.IP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# REVISED IP, in release 1.2
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; monitor [
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
use SSF.OS.Filters.IpFlowCollector
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
debug true
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
... ...&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# other attributes of the monitor
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
... ...
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ]
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ... ...&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# other attributes of IP
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ... ...
<BR>&nbsp; ]</TD>
</TR>
</TABLE></CENTER>

<P>Besides that, a configuration for the "<I>probe</I>" protoclSession
is also needed.
<CENTER><TABLE BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<TR>
<TD>&nbsp;ProtocolSession [
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name probe use SSF.OS.ProbeSession&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# in release 1.2
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; file "flow_data/flows.dat"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# specify the filename prefix of dumped data
<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; stream ipnetflow&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# stream name
<BR>&nbsp;]</TD>
</TR>
</TABLE></CENTER>
Another thing to point out: After the stream name is specified, the ProbeSession will grab it, append it with "alignment" string. So in the NetFlow data file, the name of the stream will be "ipnetflow.0" instead of "ipnetflow" specified with "stream". (Assume that the alignment is "0".)

<H3>
<A NAME="Examples and tests"></A>Examples and tests</H3>
More <A HREF="examples.html">examples and test document</A> will be added
soon.
<H3>
<A NAME="Download"></A>Download</H3>
<P>The source file of <A HREF="http://www.cs.dartmouth.edu/~yuanyg/istswork/download/NetFlow_1.2.5.tar.gz">SSF.OS.NetFlow (Including SSF.OS.NetFlow.Filter)</A>. (The name of the package is subject to change.) It should be extracted under SSF.OS, and it will generate the package SSF.OS.NetFlow.</P>

<P>The test sub directory has several tests and examples. Please get into tests directory and type "make" to see all the targets.</P>

<H3>
<A NAME="References"></A>References</H3>
<A NAME="Ref[1]"></A>[1] <A HREF="http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/nfc/nfc_3_0/nfc_ug/nfcform.htm">Cisco
NetFlow Export Datagram Format</A>
</BODY>
</HTML>
